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LEARNING BY DOING CAREER ACADEMIES 



I. Introduction 

This paper's title 1 summarizes our two main aims. For a number of years, we 
have been helping high schools and districts that are attempting to create or improve 
career academies. Our assistance includes developing the schools' own capacity to keep 
track of results for students, relying mainly on information that is ordinarily available 
from student transcripts. We believe it is essential for schools themselves to 
continuously gauge results of career academies (or other educational programs), because 
even if career academies (or other programs) have been found to be effective 
somewhere, they are unlikely to be effective everywhere. Therefore, one purpose of this 
paper is to describe how schools can self-monitor, and what they can learn in the 
process of operating career academies. Second, we wish to add to general knowledge 
about career academies, by testing some ideas about why they may be more effective in 
some places and times than in others. Specifically, we conduct a cross-site analysis to 
explore whether sites that implement certain features of the academy model to a greater 
degree also obtain bigger gains in performance of academy students. Both parts of the 
analysis in this paper focus on student performance during high school, not post-high 
school outcomes such as employment or college attendance. 

Longevity, and solid evidence of effectiveness, distinguish career academies 
from other programs, practices, and policies that were bundled under the "school-to- 
work" or "school-to-career" label in the U.S. during the 1990s. The first known career 

1 Readers who are unfamiliar with career academies should not construe the title to imply that 
career academies themselves are designed to substitute "doing" for "learning." To the contrary, 
as described in the text, most coursework in career academies is in academic subjects, and the 
intent is to prepare students for college. 
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academies were started in 1969 — well before more recent initiatives like tech-prep or 
job shadowing, though much later than co-op or vocational education (Stern et al. 1995). 
Relatively persuasive evidence of career academies' effectiveness comes from several 
quasi-experimental studies, and especially from a random-assignment evaluation 
conducted by MDRC. Section II of this paper spells out what we mean by a career 
academy, and summarizes some of the history and existing research. The paper by Orr 
and colleagues in this volume provides additional information about the purposes and 
pedagogical practices of career academies, especially those associated with the National 
Academy Foundation. 

Despite evidence that career academies can produce positive results for students, 
success is never automatic. Local circumstances may enhance or undermine the 
effectiveness of a particular career academy, within a particular school at a particular 
time. The career academy model has several elements, all of which require planning 
and effort to put in place and keep in place. In 1997 we established a Career Academy 
Support Network (CASN) to help high schools and districts that want to develop and 
improve career academies. CASN provides professional development, on-site coaching, 
and other kinds of implementation assistance. 

As part of its service, CASN uses available data from student transcripts to 
monitor implementation, and to determine whether academy students are improving 
more or less rapidly than non-academy students in the same high schools. Because of 
non-random selection of students into academies, and non-random attrition, this 
analysis is not intended to determine whether academy participation causes greater 
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improvement in students' performance. 2 But the available data can nevertheless inform 
academy stakeholders about the kinds of students who are participating and how their 
performance is changing over time. Sections III and IV illustrate this kind of analysis. 

The second question we address in this paper is whether academy students' 
growth, relative to non-academy students in the same school, is associated with 
implementation of certain key features of the model. We focus on measures having to 
do with scheduling and course-taking, because these are considered fundamental to the 
academy design, and they can be computed from student transcripts. This analysis, 
using a hierarchical model, is reported in sections V and VI. Although the differences 
between academy and non-academy students may be attributable in part to non-random 
selection or attrition, this exercise demonstrates a new procedure for estimating how 
much implementation matters. 

II. Career academies: definition, history, and research results 3 

Teachers and community groups in Philadelphia first put the essential elements 
of contemporary career academies into practice in 1969. For a decade or more this 
structure and related practices were known as the Philadelphia academy model. Before 
elaborating on the definition and history, we note that teachers had an indispensable 
role in creating the first such ventures. The subsequent spread and development of this 
approach also have depended on teachers' initiative. But teachers have seldom been 
involved in analyzing data on academies. Part of CASN's purpose is to equip the 
teachers who are responsible for academies to participate in monitoring outcomes for 
students. 

2 This paper also does not attempt to test whether CASN's efforts are producing positive results. 

3 

Some text in this section is from Stern(2003) or Stern and Wing (2004). 
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The term "career academy" was coined by Stern, Raby, and Dayton (1992) to 
describe the kind of high school configuration that originated in Philadelphia, then 
spread to California, New York City, and eventually nationwide. In 1981 the academy 
idea was exported from Philadelphia to California, starting with two academies near 
Silicon Valley. A series of evaluations (ending with Reller 1987) found improved student 
performance, and spurred California to pass legislation in 1984 supporting 10 
replications of the model. The number of state-supported academies eventually grew to 
about 300 in 2004. Also in the 1980s, New York City created the first Academies of 
Finance, sponsored by the American Express Company, which subsequently joined with 
many other companies to create NAF (National Academy Foundation). The foundation 
added the travel and tourism field in 1987, public service in 1990, and information 
technology in 1999. As of 2004, NAF supported about 400 academies in about 30 states. 
The paper by Orr and colleagues in this volume provides additional information about 
NAF, and reports results from several NAF academies. 

In addition to the Philadelphia, California, and NAF academies, many other 
academies have sprung up independently, and new organizations have sprung up to 
support them (e.g., the National Career Academy Coalition). Common themes for career 
academies are health, business and finance, arts and communications, computers, 
engineering, law and government. 

There is no authoritative, uniform definition of a career academy, and as the term 
has become popular the variation among programs that call themselves career 
academies has increased. 4 In 1993 MDRC began the first random-assignment evaluation 

4 The state of California provides grants to school districts for "partnership academies" which are 
defined by statute, but this definition does not apply to the hundreds of academies in California 
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of career academies (Kemple and Rock 1996). MDRC abstracted three main features to 
define a career academy: 

(1) School-within-a-school organization in which academy students at each grade 

level take a set of classes together, and stay with the same small group of 

teachers for at least two years. 

(2) Curriculum that includes academic courses meeting college entrance 

requirements, along with technical classes, all related to the academy theme. 

(3) Employer partnerships to provide internships and other experiences outside 

the classroom, related to the academy theme. 

The Philadelphia academies began with a focus on dropout prevention and 
vocational preparation, but they soon evolved to include preparation for 4-year colleges 
and universities. The NAF academies have been college oriented since their inception. 
Our impression from the field is that most career academy teachers explictly aim to 
prepare students for both college and careers. 

Until the 1990s, career academies existed only as separate, small units within 
larger high schools. For example, a typical career academy might serve 200 students in a 
school containing 2,000. Since the mid-1990s, however, some schools have converted 
themselves entirely into groups of career academies or of various small learning 
communities (SFCs), some of which are career academies. Fee, Ready, and Johnson 
(2001) conducted an informal national canvass to identify high schools where all 
students and teachers belonged to small learning environments. They identified 55 such 

that do not receive state funding. A few other states also have funded such academies. The 
federal School-to-Work Opportunities Act in 1994 included career academies on a list of seven 
"promising practices," but did not not define them. Building on the MDRC definition, the Career 
Academy Support Network (http: / / casn.berkeley.edu) helped negotiate a common definition 
among several networks currently promoting career academies. 
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schools, 80 percent of which were using career academies as the model. The Talent 
Development High School model developed by CRESPAR (Center for Research on the 
Education of Students Placed at Risk) is an example; every student in grades 10-12 
belongs to a career academy. On the other hand, Chicago now uses the term career 
academies to denote entire high schools that have, or did have, a vocational focus — 
these may not fit the definition we are using. 

Research evidence on career academies . Six different researchers or research 
teams, using longitudinal data from different sets of academies between 1985 and 2000, 
compared performance of academy and non-academy students from the same high 
schools. These studies are described and summarized by Stern (2003). None of these 
studies assigned students at random to the academy or non-academy groups. Most 
researchers used regression techniques to control for some observed differences between 
academy and non-academy students. In some studies the researchers and teachers 
chose each comparison student individually to match one of the academy students. 

All of these studies reported academy students performing significantly better 
while in high school. Salient findings are as follows: 

• Reller (1984, 1985) studied the first two career academies in California. Academy 
students earned significantly more course credits than a matched comparison group 
from the same high schools. One-year attrition rates ranged from 2 to 6 percent in 
academies, 10 to 21 percent in the comparison group. 

• Stern et al. (1988, 1989) studied 10 state-funded academies in California. Academy 
students tended to perform significantly better than matched comparison groups from 
the same high schools in attendance, credits earned, average grades, and likelihood of 
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staying in school. The three-year attrition rate for the cohort entering in 1985 was 7.3 
percent from academies, 14.6 percent from the comparison group. 

• Hayward and Talmadge (1995) evaluated 10 different programs that used some form 
of vocational education to promote success in high school. Two of the sites were career 
academies. These showed generally better results than other programs, improving 
students' attendance, credits, grades, and likelihood of completing high school. 

• McPartland et al. (1996, 1998) reported on the reorganization of Patterson High School 
in Baltimore in 1995, which included creation of career academies for all students in 
grades 10-12. Attendance in the first implementation year rose from 71 to 77 percent, 
contrasting with districtwide decline from 73 to 70 percent in grades 9-12. A survey of 
teachers found great improvement in reported school climate. 

• Maxwell and Rubin (1997, 2000) analyzed 1991-95 school records for three cohorts of 
students in grades 10-12 in an urban district, including nine career academies. 
Controlling for other characteristics of students, academy students received significantly 
better grades than non-academy students, in both academy and non-academy classes. 

92 percent of the academy students graduated from high school within the study period, 
compared to 82 percent of non-academy students. 

• Elliott, Hanser, and Gilroy (2000) compared 1994-96 data from three Junior Reserve 
Officers Training Corps (JROTC) career academies in large cities with student data from 
other career academies or magnets in the same or similar schools, and data on JROTC 
students not in academies and on students not in any academy or magnet program. 
Propensity-weighted regression was used in an attempt to offset possible selection bias. 
Students in JROTC career academies and in other career academies or magnets generally 
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received higher grades, had better attendance, completed more credits, and were less 
likely to drop out than statistically similar students not in academies. 

Three of these studies also gathered data on post-high school outcomes. Salient 
findings were: 

• Reller (1987) surveyed students 15 months after graduation. She found 62 percent of 
academy graduates were enrolled in postsecondary education, compared with 47 
percent of the comparison group. Expecting to complete a bachelor's degree or more 
were 55 percent of academy graduates, 22 percent of the comparison group. Another 
survey, 27 months after graduation, found no significant differences between academy 
and comparison students in employment status, wages, or hours worked. 

• Stern, Raby, and Dayton (1992) conducted follow-up surveys 10 and 22 months after 
graduation. They found no consistent differences between academy and comparison 
graduates in postsecondary attendance or degree aspirations. Academy graduates on 
average were working 3 more hours per week, but there was no consistent difference in 
hourly earnings. 

• Maxwell and Rubin (1997, 2000) found no significant differences in wages or hours 
worked between former academy and non-academy students, though former academy 
students more often said their high school program had prepared them well for further 
education and work. However, participation in postsecondary education was higher 
among former academy students: 52 percent were attending four-year colleges, 
compared to 36 percent of the non-academy students. In a subsequent study of students 
at a public university campus. Maxwell (2001) found that academy graduates were less 
likely to need remedial coursework, and were more likely to complete bachelor's 
degrees, compared to statistically similar graduates from the same district. 
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In sum, different researchers studying different sets of career academies have 
consistently found that academy students are outperforming their non-academy 
schoolmates on various measures of academic success while in high school. Post-high 
school differences in further education and employment have been less consistent, but 
where significant differences have been found they have favored the academy students. 

Although these studies statistically controlled for measured differences between 
academy and non-academy students, the possibility remains that unobserved 
differences account for the results. For example, the academy students may have been 
more motivated or better organized to begin with, and that may explain why they did 
better. Indeed, there might be a systematic tendency for more motivated or better- 
organized students to join career academies. 

The classic method for eliminating bias due to selection or self-selection is for 
researchers to use a random procedure to assign some subjects to the "treatment," and 
the others to the control group. This procedure ensures that the average difference 
between the two groups will be negligible, given a large enough sample. Despite this 
great advantage, there are well-known drawbacks and limitations to using random 
assignment in educational field studies. Denying a beneficial treatment to a control 
group may raise ethical questions. Some important educational variables — e.g., 
completing high school — cannot be experimentally manipulated. Even when random 
assignment is feasible and ethical, absence of a placebo means that Hawthorne effects 
and other biases may influence the result. Nevertheless, a well-designed random- 
assignment study can eliminate the problem of selection bias that plagues much 
educational research. (For a fuller discussion, see Stern and Wing 2004.) 
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In 1993 MDRC began its random-assignment evaluation of career academies, 
initially with ten sites, but one academy ceased operating. All nine remaining academies 
are in high schools with large proportions of low-income and minority students. Each 
was the only career academy in its high school. 

At the start of the MDRC evaluation, the academies recruited more applicants 
than they could accommodate. Applicants knew they might not be admitted. MDRC 
randomly assigned about two-thirds of the applicants to the academy; the others became 
the control group. For more than ten years since the evaluation began, MDRC has 
collected student records, surveyed students during each of their high school years, and 
conducted follow-up surveys one year and four years after high school. 

During the high school years, the career academies studied by MDRC produced 
several positive impacts on students' experience and achievement. Compared to the 
control group, academy students reported receiving more support from teachers and 
from other students (Kemple 1997). They were more likely to combine academic and 
technical courses, engage in career development activities, and work in jobs connected to 
school (Kemple, Poglinco, and Snipes 1999). As of spring of senior year, academies 
retained a larger fraction of the students whose initial characteristics made them more 
likely to drop out (Kemple and Snipes 2000). Among students at less risk of dropping 
out, academies increased participation in technical courses and career development 
activities without reducing academic course credits (Kemple and Snipes 2000). 

The first follow-up survey, one year after scheduled graduation, found no 
significant impacts on students' high school completion, GED acquisition, or 
participation in postsecondary schooling. It also showed no significant impact on 
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employment or earnings, though students who had been assigned to career academies 
were working and earning somewhat more than the control group (Kemple 2001). 

MDRC's most recent follow-up, about four years after scheduled graduation 
from high school, found large and significant impacts on employment and earnings, and 
no difference in educational attainment (Kemple and Scott-Clayton 2004). In the full 
sample, students assigned to career academies earned higher hourly wages, worked 
more hours per week, had more months of employment, and earned about 10 percent 
more per month than the control group. All these differences occurred for both males 
and females, but they were not statistically significant for females. Impacts on high 
school completion or postsecondary education were not significantly positive or 
negative for the sample as a whole or for any subgroup, but Kemple and Scott-Clayton 
(2004) note that both the academy and control groups had high rates of high school 
completion and postsecondary enrollment compared to national (NELS) data on urban 
high school students. 

III. Describing student composition and performance in a single academy 

Evaluations of career academies have found that they can produce positive 
impacts on students, but it does not follow that every academy does produce such 
impacts. The career academy model is not a sure-fire recipe for success. Much may 
depend on the particular people involved, local circumstances, and the degree of 
implementation. To know whether a career academy is yielding the desired results, 
therefore, it is necessary to keep collecting the data. In sections III-IV we illustrate the 
kind of information that can be obtained from student records. 
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Student transcript data have the great virtue of being readily available in all 
schools, but they also have serious limitations. Generally, they are messy and full of 
anomalies. For example, it is common to find students on the rolls despite having zero 
attendance for the year. We eliminated such students from our analysis. We kept 
students who had 50 or more days of attendance, even though some of these are only 
marginally involved in the school. Since state funding is determined by attendance 
counts, attendance may be one of the more accurately recorded variables. Credits 
earned, grades, suspensions, and demographic data may be subject to even greater 
inaccuracies. We weeded out anomalies we could find, but we had to take most of the 
data at face value. Despite these cautions, we think it is worth trying to find out what 
schools might be able to learn from the data in their own files. 

Are academy students a representative cross-section of the school? One of the 
first questions raised by academy teachers and their non-academy colleagues is whether 
students in the academy are different from the rest of the school. This is a contentious 
issue. Non-academy teachers would resent an academy that recruited only the school's 
"best and brightest" students. More generally, as many large high schools are now 
grouping students and teachers into smaller learning communities (SLCs), inequitable 
distribution of students among these smaller units can be cause for divisiveness that 
undermines the strategy (Muncey and McQuillan 1996). In order to know whether 
students are being assigned to academies or SLCs in some unfair way, schools are 
beginning to develop routines that enable them to monitor the data. 

For example, a series of simple bar charts (not presented here for lack of space) 
show ethnic and gender data from a biotechnology academy and its home high school, 
which we call Wyles FIS. Grade 10 is the first year of the three-year biotech academy 
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sequence. In 2002, Hispanic students were over-represented in this academy, and Asian 
students were under-represented, relative to non-academy tenth graders. But this 
imbalance disappeared a year later in grade 11. At the same time, the academy enrolled 
a disproportionately large number of females in both years. The gender imbalance may 
or may not be considered problematic, but the data at least call attention to the fact. 
Similar analysis can be done on other student characteristics that are usually recorded in 
district records, including proficiency in English, eligibility for subsidized lunch, and 
participation in special education. 

Student mobility . An important fact of life in schools is that students come and 
go. Keeping track of student mobility is difficult, but it may be a crucial part of the 
story. For instance, an academy or SLC that pushes out low-performing students would 
not be considered as effective as one that retains those students and improves their 
performance. As high schools face growing pressure to raise test scores, and as more 
high schools divide students and teachers into smaller groupings, the temptation to de- 
select low-performing students may increase. 

Table 1 illustrates how we can account for student mobility semester by 
semester. We are especially concerned about students reported as leaving the school 
district for unknown reasons. Many of these may have dropped out of school entirely, 
although others may have transferred to schools outside the district without informing 
anyone in the district where they went. The number leaving the district for unknown 
reasons therefore can be considered an upper bound on the number of dropouts. 5 
Dividing that number by the number of students who ever enrolled gives an upper 

5 Students who did transfer to other schools are also at greater risk of not completing high school. 
See Rumberger and Larson (1998). 
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bound for the dropout rate. The number ever enrolled includes students who were 
there at the beginning of the period plus those who transferred in. During the two years 
covered in Table 1, no biotech academy students left the district for unknown reasons, 
but a total of 198 (= 107 + 91) non-academy students did. These 198 are 32 percent of the 
non-academy students ever enrolled in this cohort. 

Table 2 summarizes results of similar computations for several other academies 
(school names are pseudonyms). In most of those cohort comparisons, academies had 
lower proportions of students leaving the district for unknown reasons, compared to 
non-academy students at the same schools and grade levels. In many cases the 
differences are quite large. Academies, like other SLCs, are intended to encourage 
stronger ties among students, and between students and teachers, based in part on their 
shared interest in the academy theme. The results in Tables 1 and 2 may reflect the 
holding power of those stronger relationships and shared interests. On the other hand, 
it is also possible that the academies initially selected students who were already more 
likely to stay in school. 6 

Year-to-year changes in attendance, credits, and grades . Whether students stay 
in school is an important outcome in its own right. In addition, differences in the rate of 
attrition between academy and non-academy students also affect the interpretation of 
other results. On average, students who leave the district for unknown reasons are 
probably not doing as well in high school as students who stay, so comparisons of 
performance trends between academy and non-academy students are probably biased 

6 The possibility that academies select students with lower propensities to move is also consistent 
with the finding in Table 1 that the academy accepts few transfers from outside the school. On 
the other hand, the academy did take a substantial number of transfers from the rest of the 
school. 
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against the academy. In other words, performance trends for the academy would look 
better if more low-performing students left the academy. As we compare trends over 
time in various student outcomes, it is important to keep this possible attrition bias in 
mind. 

Comparing academy students' performance with their own previous 
performance, and making that same comparison for non-academy students at the same 
grade levels, is a simple but instructive procedure schools can use to gauge an 
academy's results over time. Attendance, credits earned, and grades are three outcomes 
that schools measure periodically and record in students' transcripts. For example. 

Table 3 shows average attendance and credits earned during tenth grade by the cohort 
of academy and non-academy students. Table 3 also shows attendance and credits 
earned by those same students during ninth grade, before the biotech academy students 
had entered that program. Both attendance and number of credits declined during tenth 
grade, for both academy and non-academy students. The biggest drop was in credits 
earned by the non-academy group. Academy students declined slightly more than non- 
academy students in attendance, but slightly less in credits earned. Again, if attrition 
among academy students had been the same as among non-academy students, these 
average performance measures for academy students at the end of grade 10 would 
probably have been higher. In other words, it is likely that holding on to more students 
lowered the academy's averages. 

IV. Do academy students perform better, controlling for prior performance and other 
factors? 
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It is apparent from Table 3 that students who entered the biotech academy in 
grade 10 had compiled better records of attendance and credits earned in grade 9, 
compared to non-academy students at the school. Including students' prior 
performance in the analysis is necessary to gauge whether academy students are doing 
better than expected, compared to non-academy students. In addition, the analysis also 
can control for other student characteristics. 

Table 4 shows results from a series of regressions predicting cumulative GPA at 
the end of grade 10 for these same cohorts of academy and non-academy students at 
Wyles HS. In model 1, the only predictor is GPA at the end of grade 9. As expected, the 
coefficient on grade 9 GPA is positive and highly significant, and this remains true in 
subsequent models. Model 2 adds demographic characteristics to the list of predictors: 
gender, age, and whether the student is African American, Asian, or Hispanic (whites 
are the comparison group). Only the coefficient on gender is significant, and it indicates 
that males are receiving lower grades, controlling for the other predictors in the model. 
The coefficients on demographic characteristics also do not change much in subsequent 
models. Model 3 adds a variable indicating whether a student was in the academy or 
not. A positive coefficient would mean that academy students on average are receiving 
better grades than non-academy students, holding constant the other predictors. In this 
case the academy coefficient is positive, but barely so, and it is far from significant 
statistically. Finally, model 4 adds a multiplicative interaction between the academy 
variable and grade 9 GPA. A positive coefficient here would mean students with high 
GPA at the end of grade 9 are receiving higher grades in grade 10 if they are in the 
academy. In Table 4 this coefficient turns out to be negative, but not statistically 
significant. 
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We conclude from Table 4 that students in their first year of the biotech academy 
in 2001-2002 did not show significant improvement in grades relative to non-academy 
students, on average. And this is true whether GPA in the prior year was high or low. 

We have estimated the models in Table 4 with data from various schools and 
districts. Tables 5 and 6 show coefficients on the academy variable from regressions on 
several cohorts in this and other academies, using model 3 from Table 4 to predict each 
year's GPA, attendance, credits earned, and suspensions. In predicting GPA, the 
academy coefficient is significant in five out of those 26 cases, and negative in four of 
those five. In the other 21 cases, compared to non-academy students in the same schools 
and grade levels, academy students on average are not getting higher or lower GPAs 
than would be predicted based on their prior GPAs and demographic characteristics. In 
three out of 22 regressions predicting attendance, the coefficient on academy 
participation is significant and positive. Of the 21 regressions for credits earned during 
the year, the coefficient on academy participation was significant and positive in one 
case and significant and negative in three. Finally, none of the 21 regressions run for 
suspensions revealed a significant coefficient on the academy variable. 

Our intent in sections III and IV has been to illustrate how teachers, 
administrators, and community partners can "learn by doing" career academies. 
Transcript data reveal whether academy students represent a cross-section of the school 
in terms of gender, race, ethnicity, or other characteristics. The same data also can be 
used to calculate attrition rates, and to monitor the performance of academy students 
over time, relative to non-academy students in the same school. We have seen that the 
academies analyzed here have relatively few students leaving the district for unknown 
reasons. If students who leave the district for unknown reasons tend to be lower- 
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performing, then holding on to more of these students may reduce the apparent 
effectiveness of academies on other measures. In spite of that, several of the academies 
analyzed here do show some relative gains in student attendance, which can be seen as 
another indicator of academies' holding power. 

Students' gains in attendance, credits earned, GPA, or (reducing) disciplinary 
suspensions generally do not differ significantly between these academies and the rest of 
their host high schools. In a few academies, the "value added" in terms of GPA or 
credits earned appeared to be significantly negative. Teachers in these situations would 
want to probe for explanations and, if the data are accurate, determine what corrective 
action to take. 

V. Are results related to degrees of implementation? Introducing "course metrics" 

In addition to informing teachers and other stakeholders in individual 
academies, we are also interested in questions that involve comparisons across sites. In 
particular, we would like to know whether outcomes for academy students are related 
to how thoroughly the academy model is implemented. To explore that question, we 
have devised several new measures of implementation, which this section explains. In 
the following section, we present results from a hierarchical model testing whether those 
implementation measures are associated with student outcomes. 

One of the key features of career academies, like some other SLCs, is that 
academy students at a given grade level are scheduled to take a set of classes together. 
Typically, academy sections of English, social studies, science, and a technical or lab 
class related to the academy theme are scheduled so that academy students can take 
them together. This allows teachers to design lessons, projects, and assignments that 
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connect some of the different subjects the students are taking. In contrast to the usual 
high school schedule, the intent is to create more intellectual coherence among the 
various concepts and bodies of knowledge students are learning. Having a group of 
students take a set of classes together also adds to social cohesion among students, and 
with the teachers who share responsibility for this group. 

This kind of cohort scheduling is a major departure from standard practice in 
American high schools. 7 Scheduling academy students into academy classes therefore 
requires persistent effort by the academy teachers and school administration, especially 
given the varying scheduling systems and constraints in school districts. As a result, the 
actual implementation of this scheduling design is seldom perfect. 

We have developed three "course metrics" to indicate how closely students' 
actual course-taking comes to the academy model. We usually compute these measures 
for one year at a time, but they could also be calculated by semester or for more than one 
year. 

• Course-taking : the percentage of the academy courses in which academy students 
actually enrolled, on average. For example, suppose the intended academy course 
sequence includes four classes during sophomore year, but because of scheduling 
conflicts the average academy student ended up enrolling in only three of these 
classes. Our course-taking measure would then be 3/4 = 75 percent. A low number 
here means academy students have actually experienced less of the intended 
academy curriculum. 



7 

Cohort scheduling for secondary school students is common practice in other countries. The 
individualistic, shopping-mall approach to scheduling seems to prevail only in the U.S. 
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• Course purity : the percentage of students enrolled in the average academy class who 
are actually academy students. For example, if academy classes on average have 30 
students and 24 of these are academy students, course purity would be 24/30 = 80 
percent. A low number on this index means it is more difficult for teachers to connect 
lessons or assignments from different classes, because the different classes do not 
contain the same set of students. 

• Course coverage : credits earned in academy courses as a percentage of all credits 
earned by academy students, on average. For example, suppose the academy 
schedule called for academy students to take four classes during tenth grade, and that 
this actually happened. If the total course load all students took was six classes, then 
course coverage would be 4/6 = 67 percent. This measure was first used by 
McMullan, Sipe, and Wolf (1994) to indicate how big a part the academy or SLC 
curriculum represents in the total course load. 

These three measures are independent in the sense that knowing two of them 
does not allow you to compute the third. Conceptually, low course coverage together 
with high course-taking would indicate that the academy is designed to offer only a 
relatively small number of classes, but the average academy student is successfully 
scheduled to enroll in a high percentage of those classes. To take another scenario, low 
course-taking and high purity would mean that the average academy student misses out 
on a substantial part of the intended academy curriculum, but few non-academy 
students are enrolled in the academy classes. 

Although individual students do have some degree of choice about which 
courses they take, we view these course metrics primarily as the result of administrative 
structures and processes within the school, not as the outcome of individual student 
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choices. The third metric, course coverage, is limited by how many courses are included 
in the academy sequence; this is a school decision, not a student decision. The second 
metric, purity, is determined by how well-organized the school's scheduling process is - 
in some schools it can be chaotic! - and by the priority given to academy membership 
when the school's master schedule is constructed. Similarly, the first measure, academy 
course-taking, depends on the degree to which schedulers try, and are able, to avoid 
forcing academy students to choose between taking an academy core course and 
enrolling in some other course they must take. 

In our sample, all but one of the 22 academy cohorts have values within the [0.67, 
1.0] range for course-taking, [0.55, 1.0] for purity, and [0.22, 0.57] for coverage. The 
standard deviations among all 22 cohorts are 0.14, 0.19, and 0.12, respectively. These 
measures do indeed vary, but does that matter? 

VI. Results of hierarchical modeling across sites 

To test whether our measures of academy implementation are associated with 
improvement in academy student performance, we shift to a hierarchical modeling 
framework (Bryk and Raudenbush 1992). Individual students are the units of 
observation at level one. The level-one predictors are the prior year outcome for that 
student, along with gender, age, and race or ethnicity. At level two the unit of 
observation is a cohort of students who belonged to a particular academy, or to the non- 
academy part of the high school, at a particular grade level in a particular year. The 
analysis reported here involves 22 academy groups for which we have the necessary 
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data (from seven academies 8 ), and 22 non-academy groups, each for the same grade 
level, calendar year, and high school as the corresponding academy cohort. 

Tables 7-10 show results from estimating a sequence of five models for each 
outcome. The level-one regressions, not shown here, are the same as model 2 in Table 
4. 9 In each of the level-two models reported in Tables 7-10, the level-one intercept is 
treated as a random coefficient that varies among the 44 level-two cohorts. The first 
level-two model uses only one predictor: a variable indicating whether the student 
belonged to an academy or not. 

More explicitly, the level-one regression can be written as 
yijt = boj + biyij /t -i + x'ijb X/ 

where yq is the predicted value of an outcome (e.g., attendance) for the ith student in the 
jth cohort during year t; boj is the intercept for cohort j; bi is the coefficient on last year's 
outcome for that student, yip-i; and b x is a set of coefficients on a student's demographic 
characteristics Xij. 

Coefficients bi and b x do not vary across cohorts, but boj does. The level-two 
model in which the academy variable is the only predictor would be written as 
boj ~ go + giAj, 

where Aj = 1 if cohort j is in an academy, 0 otherwise; and gi is the coefficient reported 
for model 1 in Tables 7-10. 



8 The four missing cohorts are the 2001-02 grades 10 and 11 from Rollo and Blizzard. They were 
excluded from Tables 7-10 because course metrics were not available for these cohorts. 

9 Results in Tables 7-10 are not exactly comparable with Tables 5 and 6 because the samples are 
somewhat different due to missing data; and coefficients on student characteristics were free to 
vary across academies in the equations for which academy coefficients are reported in Tables 5 
and 6, but not in Tables 7-10. 
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The next three models add the course metrics, one at a time. The last model uses 



all three course metrics in addition to the academy variable itself. The course metrics are 
defined only for academies and are given a value of 0 for non-academy students. The 
level-two coefficients on the course metrics therefore tell us whether and to what extent 
student outcomes are better in academies that have higher scores on the course metrics. 

Tables 7-10 show the estimated coefficients and robust standard errors, corrected 
for clustering of students within the 22 groups. In Table 7, the academy coefficient by 
itself is a significantly positive predictor of attendance. But when course metrics are 
added, neither the academy coefficient nor the course metrics appear significant. In 
Table 8, no variable is significant in predicting credits earned. In predicting GPA, Table 
9 shows a negative coefficient on the course coverage metric, meaning that students tend 
to receive lower grades in academies where the academy curriculum represents a larger 
proportion of the total coursework. But when this is the only course metric in the 
model, this negative association is partly offset by a positive coefficient on the academy 
variable itself. We do not have an explanation for this finding, other than randomness in 
the data. Finally, Table 10 suggests that being in an academy may be associated with 
fewer suspensions — a positive result — but none of the course metrics is significant. 

On the whole. Tables 7-10 do not reveal any consistent association between our 
measures of academy implementation and outcomes for students. However, it would be 
premature to conclude from these results that implementation makes no difference. We 
have measured only a few implementation variables that we could calculate from 
student transcript data, and these may not be the most important. In 2004 several 
organizations that provide technical assistance to academies published a set of standards 
containing ten key elements, each with three to five separate components, amounting to 
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38 distinct practices in all. 10 Some of these other practices, or interactions among them, 
may be more important than the three indicators we were able to calculate. 

Furthermore, even the three implementation variables we did measure might 
have more explanatory power if we had data on a larger number of academies. More 
observations would give us more precise estimates, as would including academies with 
a wider range of values on our course metrics. Differences in the range we observed may 
not matter, but bigger differences might. 

Finally, the effectiveness of a given career academy may depend less on the 
absolute degree of implementation than on the contrast between the academy and the 
rest of the school (see Kemple and Snipes 2000). A perfectly implemented career 
academy may not help students much if the rest of the high school also offers similar 
benefits as the academy, but an imperfectly implemented academy may help students a 
lot if the quality of the surrounding high school is poor. Unfortunately, we did not have 
the data necessary to compare students' experience in the academy with the rest of the 
high school. 



10 The organizations involved were the National Career Academy Coalition, National Academy 
Foundation, National Center for Education & the Economy, Southern Regional Education Board, 
Center for Research on the Education of Students Placed at Risk, and CASN. The standards are 
available at each organization's web site, e.g., 
http: / / casn.berkeley.edu / resources / national_standards.html 
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VII. Conclusions and cautions 



This paper analyzed data collected as part of an ongoing project to develop and 
improve career academies in various high schools and districts. We presented two 
kinds of analysis: single-site and cross-site. The purpose of the single-site analysis is to 
inform the efforts of individual career academies. The cross-site analysis explored 
whether better-implemented academies showed bigger gains in the performance of 
academy students, relative to their non-academy counterparts. 

Section II placed this analysis in the context of earlier studies, which have found 
fairly consistent and persuasive evidence of positive effects for career academy students 
at different times and places. However, the fact that academies have produced positive 
results in some places does not imply that such results are inevitable or automatic. 
Implementing the career academy model is not easy. That is why several organizations 
and projects, including CASN, offer help to schools that request it (see footnote 10). As 
part of CASN's service, we collect student transcript data and report findings back to the 
teachers, administrators, and others involved. Sections III and IV presented examples of 
the kind of analysis we give back to schools, to inform the self-guidance of individual 
academies. For instance, comparing the gender and racial/ ethnic composition of 
academy students with non-academy students in the same school tells the stakeholders 
whether academy students represent a reasonable cross-section of the school. 

The most substantive finding in sections III-IV is that most individual academies 
have fewer students who leave the district for unknown reasons, compared to the non- 
academy parts of their high schools. This suggests that academies tend to have stronger 
holding power, a finding that is consistent with previous research. However, unlike 
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previous studies, our data on 26 different academy cohorts do not show academy 
students generally receive higher grades than their non-academy peers, when we control 
for prior grades and demographic variables. Nor do we find relative improvement 
among academy students in terms of credits earned or disciplinary suspensions during 
the year. As explained in section III, the stronger holding power of academies probably 
exerts a downward influence on measures of academy students' relative gains. In spite 
of that, we do find some evidence that academy students improve relative to non- 
academy students in terms of attendance. This seems consistent with the idea that 
academies have stronger holding power than the rest of the school: academy students 
are less likely to leave for unknown reasons, and they show bigger year-to-year gains in 
the number of days they actually come to school. 

In addition to reporting results that may be used to inform individual academies, 
we also did a cross-site analysis to explore whether academies that are better 
implemented tend to show bigger gains in student performance. For that analysis we 
had to construct measures of implementation. The most innovative piece of this paper is 
the development of three implementation measures indicating (1) what percentage of 
the designated academy courses the average academy student actually takes, (2) what 
percentage of students in the average academy class are actually academy students, and 
(3) what percentage of the average academy student's total course credits are earned in 
academy classes. We call these course-taking, purity, and coverage. Results from a 
hierarchical model did not find significant associations between these measures and 
student performance. However, we do not view this result as conclusive, because we 
measured only a few aspects of implementation, had data for only a small sample of 
academies, observed a fairly narrow range of variation in our implementation measures. 
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and could not assess the quality of academy students' experience relative to the rest of 
the school, which may be more important than the absolute degree of implementation. 

We will end with two cautionary observations. First, the information in school 
files is not always accurate. It is not collected for research purposes, and may not be 
checked or audited for accuracy. For instance, we had to exclude from the analysis some 
students who appeared in the data files despite having zero attendance for the year. 

A particularly vexing problem is how to know exactly which students are in an 
academy during any given term. Accurately identifying academy and non-academy 
students in the data base is essential for comparing their performance, but schools and 
districts do not have routine procedures to flag academy students in their data bases. As 
foundations, along with the federal and state authorities, induce more high schools to 
regroup students and teachers into academies or other small learning communities 
(SLCs), accountability pressures will require schools and districts to institute such 
procedures for identifying students by academy or SLC. Concretely, this means that at 
the the beginning and end of each term the academy or SLC leader must give the district 
a list of students. In addition, to construct the kind of course metrics we developed here, 
it is also necessary to flag every course, or every section of a course, that is designated 
for students from a particular academy or SLC. 

Our second caution has to do with implementation standards. Many school- 
reform organizations, funders, government agencies, and researchers publish advice 
about how to improve student performance. Unfortunately, too little of that advice is 
based on firm evidence. Section VI illustrated one way to test whether particular aspects 
of implementation are associated with results for students. This kind of analysis is only 
a beginning. In the particular case of career academies, most previous evaluations have 
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treated an academy as a "black box," offering little or no data on the mechanisms or 
processes through which academies may affect student performance, or the influence of 
social, historical, cultural, or institutional context. The MDRC evaluation is an exception 
(e.g., see Kemple and Snipes 2000); the paper by Orr and colleagues in this volume also 
measures a range of outcomes related to different features of the academy. More studies 
of this kind would be highly desirable. 

Obtaining desired results from career academies, other kinds of small learning 
communities, or generally any kind of reform requires a better understanding of how, 
and under what conditions, a given strategy or approach actually works. Until much 
more such analysis has been done, researchers and policy makers should be humble 
about telling schools how to improve. 
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Table 1 

Accounting for student mobility at Wyles HS 
from beginning of Grade 10, 2001-2002 to end of Grade 11, 2002-2003 





Biotech 

Academy 


Non- 

academy 


School 

Total 


No. students at beginning of Grade 10 


40 


476 


516 


Transfers in from outside school 


1 


65 




Left district for unknown reasons 




(107) 




No. students at end of first semester. 
Grade 10 


41 


434 


475 


Transfers in from outside school 




13 




Transfers from non-academy to academy 








Transfers from academy to non-academy 








Left district for unknown reasons 








No. students at end of second semester. 
Grade 10 


41 


447 


488 


Transfers in from outside school 


1 


52 


53 


Transfers from non-academy to academy 


10 


(10) 




Transfers from academy to non-academy 


(2) 


2 




Left district for unknown reasons 




(91) 




No. students at end of first semester. 
Grade 11 


50 


400 


450 


Transfers in from outside school 




7 




Transfers from non-academy to academy 








Transfers from academy to non-academy 








Left district for unknown reasons 








No. students at end of second semester. 
Grade 11 


50 


407 


457 
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Table 2 

Students leaving the district for unknown reasons, 
as a percentage of students ever in the academy or non-academy group, 
for various schools and cohorts 



School and cohort 


Academy 


Non-Academy 


Bantam HS Gr9, 1999-2000 to Grll, 2001-2002 




Biotech 

4.0 


18.7 


Bantam HS Gr9, 2000-2001 to GrlO, 2001-2002 


Biotech 

4.4 


2.1 


Bantam HS Gr9, 2001-02 to GrlO, 2002-03 


Maritime 

15.4 


Biotech 

0.0 


12.6 


Bantam HS Grll, 2001-02 to Grl2, 2002-03 




Biotech 

3.5 


8.0 


Bantam HS GrlO, 2001-02 to Grll, 2002-03 


Biotech 

10.9 


16.1 


Blizzard HS Grll, 2001-02 to Grl2, 2002-03 


Media 

10.0 


41.0 


Ohio HS Gr9, 2000-2001 to GrlO, 2001-2002 


Info Tech 
0.0 


29.3 


Ohio HS GrlO, 2001-02 to Grll, 2002-03 


Info Tech 
9.1 


34.3 


Pierce HS Gr9, 1999-2000 to Grll, 2001-2002 




Public Service 
8.5 


21.5 


Pierce HS Gr9, 2000-2001 to GrlO: 2001-2002 


Construction 

4.2 


Public Service 
0.0 


9.2 


Rollo HS GrlO, 2001-02 to Grll, 2002-03 




Teaching & Learning 
17.7 


26.5 


Rollo HS Grll, 2001-02 to Grl2, 2002-03 




Futures 

15.0 


13.5 
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Table 3 



Attendance and credits earned by Wyles HS academy and non-academy 
students in grade 10, 2001-2002, compared to previous year 





Grade 9, 2000-01 


Grade 10, 2001-02 


Ave % attendance, biotech academy 


96.9 


96.1 


Ave % attendance, non-academy 


96.4 


96.0 


Ave credits earned, biotech academy 


4.93 


4.83 


Ave credits earned, non-academy 


4.68 


4.33 



Table 4 

Regressions predicting cumulative GPA at end of grade 10, 2001-2002, 



Wyles HS (**p<.01, *p<.05) 





Model 1 


Model 2 


Model 3 


Model 4 




Coeff. 


Std 


Coeff. 


Std 


Coeff. 


Std 


Coeff. 


Std 


Variables 




Err. 




Err. 




Err. 




Err. 


Constant 


.450** 


.099 


.814 


1.288 


.795 


1.291 


.793 


1.293 


GPA Grade 9 


.755** 


.036 


.672** 


.042 


.670** 


.042 


.670** 


.044 


Male 






-.161* 


.076 


-.159* 


.077 


-.159* 


.077 


Age 






-.0082 


.081 


-.0071 


.081 


-.007 


.081 


AfriAmerican 






-.211 


.251 


-.214 


.252 


-.214 


.252 


Asian 






.235 


.160 


.239 


.160 


.238 


.160 


Hispanic 






-.070 


.151 


-.071 


.151 


-.071 


.152 


Academy 










.047 


.127 


.060 


.398 


Academy X 
GPA Grade 9 














-.005 


.136 
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Table 5 



Coefficients on academy variable from regressions 
predicting GPA and attendance, 
various academies and cohorts (**p<.01, *p<.05) 



Academy, Cohort, and Year 


GPA 


Attendance 


Coeff. 


St Err 


Coeff. 


St Err 


Ohio Info Tech Grade 10, 2001-2002 


.033 


.056 


3.505* 


1.499 


Ohio Info Tech Grade 10, 2002-2003 


.054 


.056 


.276 


1.731 


Bantam Biotech Gr 10, 2002-2003 


-.008 


.035 


.890 


.682 


Bantam Biotech Gr 11, 2002-2003 


.028 


.046 


.516 


.890 


Bantam Biotech Gr 12, 2002-2003 


-.036 


.028 


.140 


1.339 


Bantam Public Service Gr 10, 2001-2002 


.107* 


.043 


2.012* 


1.005 


Bantam Public Service Gr 11, 2001-2002 


-.090** 


.000 


-.535 


.810 


Bantam Public Service Gr 10, 2002-2003 


-.002 


.038 


.674 


.860 


Bantam Public Service Gr 11, 2002-2003 


-.067** 


.034 


-.478 


1.021 


Bantam Public Service Gr 12, 2002-2003 


-.010 


.023 


1.621 


1.343 


Bantam Construction Gr 10, 2001-2002 


-.011 


.058 


-1.151 


1.279 


Bantam Construction Gr 10, 2002-2003 


-.114* 


.050 


-1.290 


1.172 


Bantam Construction Gr 11, 2002-2003 


-.064* 


.031 


-.134 


1.298 


Wyles Biotech Gr 10, 2001-2002 


.047 


.127 


-.122 


.787 


Wyles Biotech Gr 12, 2001-2002 


.006 


.119 


1.533 


1.027 


Wyles Biotech Gr 10, 2002-2003 


.103 


.119 


-1.049 


.613 


Wyles Biotech Gr 11, 2002-2003 


.056 


.111 


.136 


.631 


Wyles Biotech Gr 12, 2002-2003 


.066 


.120 


.943 


.704 


Rollo Teaching & Learning Gr 10, 2001-2002 


-.033 


.046 






Rollo Teaching & Learning Gr 11, 2001-2002 


-.018 


.026 






Rollo Teaching & Learning Gr 11, 2002-2003 


-.035 


.045 






Rollo Teaching & Learning Gr 12, 2002-2003 


-.018 


.026 






Blizzard Media Gr 10, 2001-2002 


.034 


.067 


3.212 


1.687 


Blizzard Media Gr 11 2001-2002 


.016 


.032 


1.977 


1.161 


Blizzard Media Gr 11, 2002-2003 


.023 


.062 


3.801* 


1.859 


Blizzard Media Gr 12, 2002-2003 


.005 


.030 


1.350 


.807 
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Table 6 



Coefficients on academy variable from regressions 
predicting credits earned and suspensions, 
various academies and cohorts (**p<.01, *p<.05) 



Academy, Cohort, and Year 


Credits earned 


Suspensions 


Coeff. 


St Err 


Coeff. 


St Err 


Ohio Info Tech Grade 10, 2001-2002 


.140 


.202 


.021 


.062 


Ohio Info Tech Grade 10, 2002-2003 


-.564* 


.222 


-.031 


.073 


Bantam Biotech Gr 10, 2002-2003 


-.106 


.125 


-.062 


.038 


Bantam Biotech Gr 11, 2002-2003 


.100 


.216 


.000 


.043 


Bantam Biotech Gr 12, 2002-2003 


.336 


.352 


-.029 


.050 


Bantam Public Service Gr 10, 2001-2002 


.204 


.174 


.026 


.030 


Bantam Public Service Gr 11, 2001-2002 


.271 


.176 


-.004 


.028 


Bantam Public Service Gr 10, 2002-2003 


.143 


.131 


-.005 


.050 


Bantam Public Service Gr 11, 2002-2003 


.261 


.200 


-.064 


.041 


Bantam Public Service Gr 12, 2002-2003 


-.036 


.239 


-.066 


.047 


Bantam Construction Gr 10, 2001-2002 


-.500* 


.227 


.017 


.038 


Bantam Construction Gr 10, 2002-2003 


-.093 


.171 


-.032 


.069 


Bantam Construction Gr 11, 2002-2003 


.241 


.260 


.015 


.057 


Wyles Biotech Gr 10, 2001-2002 


-.001 


.147 






Wyles Biotech Gr 12, 2001-2002 


-.039 


.130 






Wyles Biotech Gr 10, 2002-2003 


.460** 


.144 






Wyles Biotech Gr 11, 2002-2003 


.015 


.149 






Wyles Biotech Gr 12, 2002-2003 


.180 


.132 






Rollo Teaching & Learning Gr 10, 2001-2002 


-.351 


1.086 


-.092 


.066 


Rollo Teaching & Learning Gr 11, 2001-2002 


-2.823** 


.873 


.036 


.064 


Rollo Teaching & Learning Gr 11, 2002-2003 


-.347 


1.063 


-.094 


.059 


Rollo Teaching & Learning Gr 12, 2002-2003 






.006 


.054 


Blizzard Media Gr 10, 2001-2002 






-.073 


.055 


Blizzard Media Gr 11 2001-2002 






-.061 


.048 


Blizzard Media Gr 11, 2002-2003 






-.064 


.055 


Blizzard Media Gr 12, 2002-2003 






-.069 


.048 
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Table 7 

Level-two coefficients on academy variable and course metrics, predicting level-one 
intercept in model for attendance (*p<.05; N=6017) 



Model 


Academy 


Course-taking 


Purity 


Coverage 


Coeff. 


Std err 


Coeff. 


Std err 


Coeff. 


Std err 


Coeff. 


Std err 


1 


0.713* 


0.301 








2 


-1.493 


1.360 


2.554 


1.532 






3 


0.371 


0.947 




0.394 


1.049 




4 


0.888 


0.759 






-0.398 


1.585 


5 


-1.211 


1.565 


2.693 


1.504 


0.080 


1.204 


-1.069 


1.807 



Table 8 

Level-two coefficients on academy variable and course metrics, predicting level-one 
intercept in model for credits earned (*p<.05; N=5477) 



Model 


Academy 


Course-taking 


Purity 


Coverage 


Coeff 


Std err 


Coeff 


Std err 


Coeff 


Std err 


Coeff 


Std err 


1 


0.085 


0.104 








2 


0.546 


0.367 


-0.541 


0.449 






3 


0.162 


0.263 




-0.091 


0.291 




4 


0.388 


0.392 






-0.732 


0.909 


5 


0.679 


0.438 


-0.597 


0.548 


0.415 


0.422 


-1.054 


1.046 
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Table 9 

Level-two coefficients on academy variable and course metrics, predicting level-one 
intercept in model for GPA (*p<.05; N=6017) 



Model 


Academy 


Course-taking 


Purity 


Coverage 


Coeff 


Std err 


Coeff 


Std err 


Coeff 


Std err 


Coeff 


Std err 


1 


0.002 


0.041 








2 


-0.103 


0.089 


0.123 


0.103 






3 


0.064 


0.076 




-0.072 


0.083 




4 


0.194* 


0.072 






-0.440* 


0.178 


5 


0.006 


0.110 


0.178 


0.100 


0.116 


0.121 


-0.590* 


0.224 



Table 10 

Level-two coefficients on academy variable and course metrics, predicting level-one 



intercept in model for suspensions (*p<.05; N=4119) 



Model 


Academy 


Course-taking 


Purity 


Coverage 


Coeff 


Std err 


Coeff 


Std err 


Coeff 


Std err 


Coeff 


Std en 


1 


-0.005 


0.019 








2 


0.0276 


0.083 


-0.038 


0.087 






3 


-0.081* 


0.039 




0.087 


0.059 




4 


-0.023 


0.038 






0.043 


0.080 


5 


-0.028 


0.061 


-0.077 


0.101 


0.139 


0.120 


-0.075 


0.153 
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